Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update llvm-project to b44b3494f60296db6aca38a14cab061d9b747a0a #2511

Merged
merged 1 commit into from
Oct 17, 2023
Merged

Update llvm-project to b44b3494f60296db6aca38a14cab061d9b747a0a #2511

merged 1 commit into from
Oct 17, 2023

Conversation

AmosLewis
Copy link
Collaborator

The main purpose is to bring in the new mesh dialect change. llvm/llvm-project#68007

@AmosLewis
Copy link
Collaborator Author

AmosLewis commented Oct 12, 2023 •
edited
Loading 

 cmake --build build --target tools/torch-mlir/all       
[97/97] Linking CXX executable bin/torch-mlir-capi-torch-test
FAILED: bin/torch-mlir-capi-torch-test 
: && /usr/bin/clang++ -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -g -Wl,-rpath-link,/nodclouddata/chi/src/torch-mlir/build/./lib tools/torch-mlir/test/CAPI/CMakeFiles/torch-mlir-capi-torch-test.dir/torch.c.o -o bin/torch-mlir-capi-torch-test  -Wl,-rpath,"\$ORIGIN/../lib"  -lpthread  lib/libMLIRCAPIIR.a  lib/libMLIRCAPIRegisterEverything.a  lib/libTorchMLIRCAPI.a  lib/libMLIRCAPIIR.a  lib/libMLIRAffineTransformOps.a  lib/libMLIRAMDGPUTransforms.a  lib/libMLIRArithValueBoundsOpInterfaceImpl.a  lib/libMLIRAsyncTransforms.a  lib/libMLIRBufferizationPipelines.a  lib/libMLIRBufferizationTransformOps.a  lib/libMLIRControlFlowTransforms.a  lib/libMLIRFuncTransformOps.a  lib/libMLIRGPUTransformOps.a  lib/libMLIRIRDL.a  lib/libMLIRMemRefTransformOps.a  lib/libMLIRMeshDialect.a  lib/libMLIRMLProgramTransforms.a  lib/libMLIRNVGPUTransformOps.a  lib/libMLIRNVGPUTransforms.a  lib/libMLIRSCFTransformOps.a  lib/libMLIRShapeOpsTransforms.a  lib/libMLIRSparseTensorPipelines.a  lib/libMLIRSparseTensorTransformOps.a  lib/libMLIRLinalgTransformOps.a  lib/libMLIRSparseTensorTransforms.a  lib/libMLIRSparseTensorUtils.a  lib/libMLIRSPIRVModuleCombiner.a  lib/libMLIRSPIRVTransforms.a  lib/libMLIRTensorInferTypeOpInterfaceImpl.a  lib/libMLIRTensorTransformOps.a  lib/libMLIRTransformPDLExtension.a  lib/libMLIRTransformDialectTransforms.a  lib/libMLIRVectorTransformOps.a  lib/libMLIRTransformDialect.a  lib/libMLIRTransformDialectUtils.a  lib/libMLIRTargetCpp.a  lib/libMLIRSPIRVTranslateRegistration.a  lib/libMLIRSPIRVDeserialization.a  lib/libMLIRToLLVMIRTranslationRegistration.a  lib/libMLIRArmNeonToLLVMIRTranslation.a  lib/libMLIRArmSMEToLLVMIRTranslation.a  lib/libMLIRArmSVEToLLVMIRTranslation.a  lib/libMLIRAMXToLLVMIRTranslation.a  lib/libMLIROpenACCToLLVMIRTranslation.a  lib/libMLIROpenMPToLLVMIRTranslation.a  lib/libMLIRX86VectorToLLVMIRTranslation.a  lib/libMLIRFromLLVMIRTranslationRegistration.a  lib/libMLIRLLVMIRToLLVMTranslation.a  lib/libMLIRTargetLLVMIRImport.a  lib/libMLIRArithToAMDGPU.a  lib/libMLIRArmNeon2dToIntr.a  lib/libMLIRArmSMEToSCF.a  lib/libMLIRBufferizationToMemRef.a  lib/libMLIRComplexToLibm.a  lib/libMLIRComplexToLLVM.a  lib/libMLIRComplexToSPIRV.a  lib/libMLIRComplexToStandard.a  lib/libMLIRControlFlowToSCF.a  lib/libMLIRControlFlowToSPIRV.a  lib/libMLIRConvertToLLVMPass.a  lib/libMLIRConvertToLLVMInterface.a  lib/libMLIRGPUToNVVMTransforms.a  lib/libMLIRGPUToROCDLTransforms.a  lib/libMLIRAMDGPUToROCDL.a  lib/libMLIRAMDGPUUtils.a  lib/libMLIRAMDGPUDialect.a  lib/libMLIRGPUToSPIRV.a  lib/libMLIRGPUToVulkanTransforms.a  lib/libMLIRSPIRVSerialization.a  lib/libMLIRSPIRVBinaryUtils.a  lib/libMLIRIndexToLLVM.a  lib/libMLIRLinalgToStandard.a  lib/libMLIRMathToFuncs.a  lib/libMLIRMathToLibm.a  lib/libMLIRMathToLLVM.a  lib/libMLIRMathToSPIRV.a  lib/libMLIRNVGPUToNVVM.a  lib/libMLIRGPUToGPURuntimeTransforms.a  lib/libMLIRAsyncToLLVM.a  lib/libMLIRNVVMToLLVM.a  lib/libMLIROpenACCToSCF.a  lib/libMLIROpenACCDialect.a  lib/libMLIROpenMPToLLVM.a  lib/libMLIRReconcileUnrealizedCasts.a  lib/libMLIRSCFToControlFlow.a  lib/libMLIRSCFToEmitC.a  lib/libMLIREmitCDialect.a  lib/libMLIRSCFToGPU.a  lib/libMLIRGPUTransforms.a  lib/libMLIRAsyncDialect.a  lib/libMLIRIndexDialect.a  lib/libMLIRNVVMTarget.a  lib/libMLIRNVVMToLLVMIRTranslation.a  lib/libMLIRROCDLTarget.a  lib/libMLIRROCDLToLLVMIRTranslation.a  lib/libMLIRROCDLDialect.a  lib/libMLIRTargetLLVM.a  lib/libMLIRBuiltinToLLVMIRTranslation.a  lib/libMLIRGPUToLLVMIRTranslation.a  lib/libMLIRLLVMToLLVMIRTranslation.a  lib/libMLIRExecutionEngineUtils.a  lib/libLLVMPasses.a  lib/libLLVMCoroutines.a  lib/libLLVMipo.a  lib/libLLVMVectorize.a  lib/libLLVMLinker.a  lib/libLLVMInstrumentation.a  lib/libLLVMCodeGen.a  lib/libLLVMCodeGenTypes.a  lib/libLLVMHipStdPar.a  lib/libLLVMIRPrinter.a  lib/libLLVMObjCARCOpts.a  lib/libLLVMTarget.a  lib/libMLIRAffineToStandard.a  lib/libMLIRSCFToOpenMP.a  lib/libMLIROpenMPDialect.a  lib/libMLIROpenACCMPCommon.a  lib/libMLIRSCFToSPIRV.a  lib/libMLIRMemRefToSPIRV.a  lib/libMLIRShapeToStandard.a  lib/libMLIRSPIRVToLLVM.a  lib/libMLIRSPIRVUtils.a  lib/libMLIRMemRefToLLVM.a  lib/libMLIRTensorToLinalg.a  lib/libMLIRTensorToSPIRV.a  lib/libMLIRArithToSPIRV.a  lib/libMLIRFuncToSPIRV.a  lib/libMLIRTosaToArith.a  lib/libMLIRTosaToLinalg.a  lib/libMLIRTosaToSCF.a  lib/libMLIRTosaToTensor.a  lib/libMLIRUBToLLVM.a  lib/libMLIRUBToSPIRV.a  lib/libMLIRVectorToGPU.a  lib/libMLIRNVGPUUtils.a  lib/libMLIRVectorToLLVMPass.a  lib/libMLIRAMXTransforms.a  lib/libMLIRAMXDialect.a  lib/libMLIRArmNeonDialect.a  lib/libMLIRArmSMETransforms.a  lib/libMLIRArmSVETransforms.a  lib/libMLIRArmSVEDialect.a  lib/libMLIRVectorToArmSME.a  lib/libMLIRArmSMEUtils.a  lib/libMLIRArmSMEDialect.a  lib/libMLIRVectorToSPIRV.a  lib/libMLIRSPIRVConversion.a  lib/libMLIRSPIRVDialect.a  lib/libMLIRCAPITransforms.a  lib/libTorchMLIRInitAll.a  lib/libMLIRFuncAllExtensions.a  lib/libMLIRFuncInlinerExtension.a  lib/libTorchMLIRTorchConversionPasses.a  lib/libMLIRTosaTransforms.a  lib/libTorchMLIRTMTensorPasses.a  lib/libMLIRLinalgTransforms.a  lib/libMLIRMemRefTransforms.a  lib/libMLIRArithTransforms.a  lib/libMLIRFuncTransforms.a  lib/libMLIRNVGPUDialect.a  lib/libMLIRSCFTransforms.a  lib/libMLIRTensorTilingInterfaceImpl.a  lib/libMLIRLinalgUtils.a  lib/libMLIRTensorTransforms.a  lib/libMLIRAffineTransforms.a  lib/libMLIRSCFUtils.a  lib/libMLIRVectorToLLVM.a  lib/libMLIRTargetLLVMIRExport.a  lib/libMLIRLLVMIRTransforms.a  lib/libMLIRNVVMDialect.a  lib/libMLIRTranslateLib.a  lib/libLLVMFrontendOpenMP.a  lib/libLLVMScalarOpts.a  lib/libLLVMAggressiveInstCombine.a  lib/libLLVMInstCombine.a  lib/libLLVMTransformUtils.a  lib/libMLIRTensorUtils.a  lib/libMLIRX86VectorTransforms.a  lib/libMLIRFuncToLLVM.a  lib/libMLIRArithToLLVM.a  lib/libMLIRControlFlowToLLVM.a  lib/libMLIRLLVMCommonConversion.a  lib/libMLIRVectorToSCF.a  lib/libMLIRVectorTransforms.a  lib/libMLIRAffineUtils.a  lib/libMLIRBufferizationTransforms.a  lib/libMLIRGPUDialect.a  lib/libMLIRDLTIDialect.a  lib/libMLIRMemRefUtils.a  lib/libMLIRVectorUtils.a  lib/libMLIRAffineAnalysis.a  lib/libTorchMLIRConversionPasses.a  lib/libTorchMLIRTorchConversionToMLProgram.a  lib/libMLIRMLProgramDialect.a  lib/libTorchMLIRTorchToArith.a  lib/libTorchMLIRTorchToLinalg.a  lib/libTorchMLIRTorchToSCF.a  lib/libTorchMLIRTorchConversionDialect.a  lib/libTorchMLIRTorchToTMTensor.a  lib/libTorchMLIRTMTensorDialect.a  lib/libTorchMLIRTorchToTosa.a  lib/libMLIRTosaDialect.a  lib/libMLIRQuantUtils.a  lib/libTorchMLIRTorchToStablehlo.a  lib/libStablehloOps.a  lib/libTorchMLIRConversionUtils.a  lib/libMLIRLinalgDialect.a  lib/libMLIRParser.a  lib/libMLIRTilingInterface.a  lib/libChloOps.a  lib/libStablehloTypeInference.a  lib/libStablehloAssemblyFormat.a  lib/libStablehloBase.a  lib/libMLIRQuantDialect.a  lib/libStablehloBroadcastUtils.a  lib/libMLIRShapeDialect.a  lib/libTorchMLIRRefBackend.a  lib/libMLIRMathTransforms.a  lib/libMLIRMathDialect.a  lib/libMLIRSCFDialect.a  lib/libMLIRBufferizationDialect.a  lib/libMLIRSparseTensorDialect.a  lib/libMLIRControlFlowDialect.a  lib/libMLIRVectorDialect.a  lib/libMLIRTensorDialect.a  lib/libMLIRAffineDialect.a  lib/libMLIRMemRefDialect.a  lib/libMLIRShapedOpInterfaces.a  lib/libMLIRValueBoundsOpInterface.a  lib/libMLIRParallelCombiningOpInterface.a  lib/libMLIRDestinationStyleOpInterface.a  lib/libMLIRVectorInterfaces.a  lib/libMLIRMaskableOpInterface.a  lib/libMLIRMaskingOpInterface.a  lib/libMLIRX86VectorDialect.a  lib/libMLIRDialectUtils.a  lib/libMLIRArithUtils.a  lib/libMLIRComplexDialect.a  lib/libMLIRArithAttrToLLVMConversion.a  lib/libMLIRArithDialect.a  lib/libMLIRUBDialect.a  lib/libMLIRCastInterfaces.a  lib/libMLIRInferIntRangeCommon.a  lib/libMLIRLLVMDialect.a  lib/libLLVMBitWriter.a  lib/libLLVMAnalysis.a  lib/libLLVMProfileData.a  lib/libLLVMSymbolize.a  lib/libLLVMDebugInfoPDB.a  lib/libLLVMDebugInfoMSF.a  lib/libLLVMDebugInfoBTF.a  lib/libLLVMDebugInfoDWARF.a  lib/libLLVMObject.a  lib/libLLVMMCParser.a  lib/libLLVMMC.a  lib/libLLVMDebugInfoCodeView.a  lib/libLLVMIRReader.a  lib/libLLVMAsmParser.a  lib/libLLVMBitReader.a  lib/libLLVMTextAPI.a  lib/libMLIRDialect.a  lib/libTorchMLIRTorchPasses.a  lib/libTorchMLIRTorchDialect.a  lib/libMLIRBytecodeWriter.a  lib/libMLIRFuncDialect.a  lib/libMLIRBytecodeOpInterface.a  lib/libMLIRBytecodeReader.a  lib/libMLIRAsmParser.a  lib/libMLIRTransforms.a  lib/libMLIRTransformUtils.a  lib/libMLIRRewrite.a  lib/libMLIRPDLToPDLInterp.a  lib/libMLIRPass.a  lib/libMLIRAnalysis.a  lib/libMLIRCallInterfaces.a  lib/libMLIRControlFlowInterfaces.a  lib/libMLIRPresburger.a  lib/libMLIRLoopLikeInterface.a  lib/libMLIRViewLikeInterface.a  lib/libMLIRInferIntRangeInterface.a  lib/libMLIRDataLayoutInterfaces.a  lib/libMLIRPDLInterpDialect.a  lib/libMLIRPDLDialect.a  lib/libMLIRInferTypeOpInterface.a  lib/libMLIRSideEffectInterfaces.a  lib/libMLIRFunctionInterfaces.a  lib/libMLIRMemorySlotInterfaces.a  lib/libMLIRCopyOpInterface.a  lib/libMLIRRuntimeVerifiableOpInterface.a  lib/libMLIRIR.a  lib/libMLIRSupport.a  lib/libTorchMLIRTorchUtils.a  lib/libLLVMCore.a  lib/libLLVMBinaryFormat.a  lib/libLLVMTargetParser.a  lib/libLLVMRemarks.a  lib/libLLVMBitstreamReader.a  lib/libLLVMSupport.a  -lrt  -ldl  -lpthread  -lm  /usr/lib/x86_64-linux-gnu/libz.so  /usr/lib/x86_64-linux-gnu/libtinfo.so  lib/libLLVMDemangle.a && :
/usr/bin/ld: lib/libTorchMLIRRefBackend.a(RefBackend.cpp.o): in function `GeneralizePadOpPattern':
/nodclouddata/chi/src/torch-mlir/externals/llvm-project/llvm/../mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h:1295: undefined reference to `vtable for mlir::linalg::GeneralizePadOpPattern'
clang: error: linker command failed with exit code 1 (use -v to see invocation)
ninja: build stopped: subcommand failed.

It looks like it is the miss dep of MLIRLinalgTransforms for refbackend.

@AmosLewis AmosLewis force-pushed the mesh_llvm_uplift branch 3 times, most recently from 762968b to a2bc76a Compare October 16, 2023 20:34
@AmosLewis AmosLewis marked this pull request as ready for review October 17, 2023 01:16
@AmosLewis AmosLewis merged commit 14a4da9 into llvm:main Oct 17, 2023
5 checks passed
@AmosLewis AmosLewis deleted the mesh_llvm_uplift branch October 17, 2023 02:29
@sjain-stanford sjain-stanford mentioned this pull request Oct 18, 2023
Merged
sjain-stanford added a commit that referenced this pull request Oct 18, 2023
@sjain-stanford
Bump LLVM to get bazel fixes (#2517)
52abae1 
The last llvm bump in #2511
pointed to
llvm/llvm-project@b44b349,
however the bazel build upstream was not clean at this point:

```
ERROR: /root/.cache/bazel/_bazel_root/b89349c08f7224396763d14fe35cba11/external/llvm-project/mlir/BUILD.bazel:5837:18: TdGenerate
external/llvm-project/mlir/include/mlir/Dialect/LLVMIR/NVVMOpsInterface.h.inc failed: (Exit 1): mlir-tblgen failed: error executing command ...
                                                                                                                                                    
external/llvm-project/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td:20:9: error: Could not find include file 'mlir/Dialect/LLVMIR/BasicPtxBuilderInterface.td'                                                                                                           
include "mlir/Dialect/LLVMIR/BasicPtxBuilderInterface.td"                                                                                                                                                                                                              
        ^                                                                                                                                                                                                                                                              
external/llvm-project/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td:20:9: error: Unexpected token at top level                                                                                                                                                           
include "mlir/Dialect/LLVMIR/BasicPtxBuilderInterface.td"                                                                                                                                                                                                              
        ^       
```

The bazel fixes followed in a subsequent commit at
llvm/llvm-project@28b27c1.
This PR bumps LLVM by a few more commits (to include the bazel fixes)
which helps restore Torch-MLIR's bazel build back to 🟢 .

GHA workflow to test bazel build:
https://github.com/sjain-stanford/torch-mlir/actions/runs/6555101471/job/17803082508
@sjain-stanford sjain-stanford mentioned this pull request Oct 19, 2023
Merged
sjain-stanford added a commit to cruise-automation/mlir-tcp that referenced this pull request Oct 20, 2023
@sjain-stanford
[RFC + PR] Use TCP for {LLVM / Torch-MLIR / StableHLO} Green Commit S…
1852bea 
…ync (#11)

## Why
When bumping LLVM up, it is crucial to be able to test all downstream
repos depending on it to ensure they work **in tandem** (and not just in
isolation).

In the past, LLVM upgrades were simpler because torch-mlir took a hard
dependency on mhlo/stablehlo and, in doing so, ensured that the llvm
"green commit" (sha1) that torch-mlir and stablehlo were built+tested
against was pre-identified. During this time mlir-tcp was developed on a
branch of torch-mlir.

This meant when upgrades were needed downstream, we’d simply point to
torch-mlir@HEAD (sha4) and pick the llvm-project (sha1) and
mhlo/stablehlo (sha3) hashes it’d refer to, since these are already
tested to work together. This became our set of green commits
(llvm@sha1, stablehlo@sha3, torch-mlir@sha4) for downstream integrations
(e.g cruise monorepo).

<img width="500" alt="image"
src="https://github.com/cruise-automation/mlir-tcp/assets/19234106/42078522-466c-449f-8d7e-496facc1447c">

At present the situation is complicated because torch-mlir no longer
takes a hard dependency on stablehlo (stablehlo e2e tests
[disabled](llvm/torch-mlir#2460)).

Here's details from a recent upgrade scenario that motivated this RFC.

We picked torch-mlir@HEAD which was right after the llvm bump in
llvm/torch-mlir#2511 pointing to
llvm/llvm-project@b44b349,
but soon realized (when we started building torch-mlir) that the llvm
bazel build upstream was broken:

```
ERROR: /root/.cache/bazel/_bazel_root/b89349c08f7224396763d14fe35cba11/external/llvm-project/mlir/BUILD.bazel:5837:18: TdGenerate
external/llvm-project/mlir/include/mlir/Dialect/LLVMIR/NVVMOpsInterface.h.inc failed: (Exit 1): mlir-tblgen failed: error executing command ...
                                                                                                                                                    
external/llvm-project/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td:20:9: error: Could not find include file 'mlir/Dialect/LLVMIR/BasicPtxBuilderInterface.td'                                                                                                           
include "mlir/Dialect/LLVMIR/BasicPtxBuilderInterface.td"                                                                                                                                                                                                              
        ^                                                                                                                                                                                                                                                              
```

The bazel fixes followed in a subsequent commit at
llvm/llvm-project@28b27c1.
Hence llvm had to be re-bumped in torch-mlir
(llvm/torch-mlir#2517). However, after a bit
more work we hit these failing stablehlo tests, which surfaced the fact
that stablehlo pointed to by torch-mlir could no longer be used, and we
had to separately identify the sha3 of stablehlo that would build
cleanly against sha1 of llvm.

```
@stablehlo//stablehlo/conversions/tosa/tests:binary.mlir.test            FAILED in 0.7s                                                       
@stablehlo//stablehlo/tests:print_stablehlo.mlir.test                    FAILED in 4.7s
```


This meant the burden of identifying the llvm green commit (that works
across the board) is shifted further downstream from torch-mlir.
Incidentally we are in a great position to leverage mlir-tcp to identify
the set of green commits, given it already directly depends on each of
these repos.

<img width="500" alt="image"
src="https://github.com/cruise-automation/mlir-tcp/assets/19234106/cadd38c4-71ec-45b0-8888-85ac0bfd4e99">


## What
This PR is an attempt to leverage the mlir-tcp repo as our "proxy" for
such downstream integrations, and _I think_ contains everything needed
to be able to do that.

## How
Specifically, we should now be able to run these from the comfort of
`mlir-tcp`:

```shell
bazel test --config=clang_linux @llvm-project//mlir/...
bazel test --config=clang_linux @stablehlo//...
bazel test --config=clang_linux @torch-mlir//...
```

We provide `local_repos.bzl` that allows easier local testing of patches
that later need to be upstreamed, and while they're being upstreamed we
could land them as patches to our `http_archive` targets.

Note: I include a `stablehlo.patch` that allows testing stablehlo from
`mlir-tcp`. This is temporary and can be removed once
openxla/stablehlo#1810 lands.

This PR also enables each of the 3p test suites as GHA workflows
(non-merge gating for now, we can change this). These workflows are
automatically skipped unless a change is made to `deps.bzl` (which
usually means bumping 3p deps), as it would be unnecessary to run them
for every PR and `main` commit post-merge.

Here's a snapshot from this PR's workflows, having bumped stablehlo
commit.

<img width="747" alt="image"
src="https://github.com/cruise-automation/mlir-tcp/assets/19234106/e535ed39-33f7-4941-958c-3a5d0c0adef6">
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@stellaraccident stellaraccident stellaraccident approved these changes

@dan-garvey dan-garvey Awaiting requested review from dan-garvey

@vivekkhandelwal1 vivekkhandelwal1 Awaiting requested review from vivekkhandelwal1

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@AmosLewis @stellaraccident